Image processing apparatus, image processing program, electronic camera, and image processing method for image analysis of magnification chromatic aberration

ABSTRACT

An image processing apparatus includes an image obtaining unit, a color shift detecting unit, a controlling unit, and a correction amount calculating unit. The image obtaining unit obtains image data. The color shift detecting unit detects color shifts of this image data. The controlling unit determines fitting reliability of the color shift detection result and selects a magnification chromatic aberration model suitable for color shift fitting according to the reliability. The correction amount calculating unit fits the magnification chromatic aberration model selected by the controlling unit to the color shift detection result and obtains correction amounts of the magnification chromatic aberration for the image data.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation application of InternationalApplication No. PCT/JP2007/000155, filed Mar. 1, 2007, designating theU.S., in which the International Application claims a priority date ofMar. 1, 2006, based on prior filed Japanese Patent Application No.2006-054450, the entire contents of which are incorporated herein byreference.

BACKGROUND

1. Field

The present application relates to a technique for an image analysis ofmagnification chromatic aberration.

2. Description of the Related Art

Generally, in an electronic camera, it is known that a color shiftoccurs in imaged image data by magnification chromatic aberration of animaging optical system thereof. Conventionally, there have been proposedtechniques to detect the magnification chromatic aberration from thiscolor shift of image data.

For example, Patent Document 1 (Japanese Unexamined Patent ApplicationPublication No. 2002-344978) obtains magnification chromatic aberrationbetween R and G in image data by finding a minimum point in differencesbetween the R and G components while changing magnification of the Rcomponent with various magnification changing ratios.

In addition, Patent Document 2 (Japanese Unexamined Patent ApplicationPublication No. 2002-320237) obtains the magnification chromaticaberration by detecting multiple color shifts in image data and byfitting a mathematical formula model to these color shifts.

Further, Patent Document 3 (Japanese Unexamined Patent ApplicationPublication No. 2004-286684) discloses to fit a linear expression tocolor shifts in image data in a least square approach.

Usually, a magnification chromatic aberration model including a cubicexpression of an image height is used for performing precise fitting ofcolor shifts which changes complicatedly. However, when the cubicmagnification chromatic aberration model is used, there arise thefollowing problems.

(1) Easily Affected by a Color Shift Detection Error.

The color shifts detected in image data include detection errors causedby influences of a color structure specific to an object, a color noise,a false color, etc. other than the color shift caused by themagnification chromatic aberration. Meanwhile, the fitting in the cubicexpression has a high degree of freedom and easily responds sensitivelyto these detection errors. Therefore, it may be probable to obtain wrongvalues for parameters in a mathematical formula model of themagnification chromatic aberration.

(2) The Color Shift Cannot Always be Detected at a Sampling IntervalSufficient for the Image Height.

For performing the fitting correctly on a bending point or the like inthe cubic expression, it is necessary to detect the color shifts at asampling interval appropriate for the image height. In some image data,however, a flat image structure occupies a certain area and the colorshifts cannot always be detected in a sufficient sampling interval. Whenapproximation by the cubic expression is performed in this situation,there is a high probability that wrong values are obtained forparameters in the mathematical formula model of the magnificationchromatic aberration.

(3) There is a Case in which the Magnification Chromatic AberrationCannot be Approximated by the Cubic Expression of the Image Height.

In a typical tendency, the color shift by the magnification chromaticaberration is proportional to the image height in a vicinity of an imagecenter and changes nonlinearly against the image height in a peripheralimage area. Such a behavior cannot always be approximated by the cubicexpression. When approximation by the cubic expression is performed inthis situation, there is a high probability that wrong values areobtained for parameters in the mathematical formula model of themagnification chromatic aberration.

(4) There is a Case in which Anisotropic Magnification ChromaticAberration Occurs

Some lenses may cause the magnification chromatic aberration asymmetricto an optical axis. Meanwhile, the color shift detection errorsfrequently have anisotropic components. Therefore, it is difficult todiscriminate between the anisotropic magnification chromatic aberrationand the anisotropic detection errors, and there is a high probability ofobtaining wrong values for parameters in the mathematical formula modelof the magnification chromatic aberration.

SUMMARY

The proposition is to estimate the magnification chromatic aberrationappropriately from the color shift detection result of image data, inview of such problems.

An image processing apparatus is an apparatus that estimatesmagnification chromatic aberration of image data by fitting amagnification chromatic aberration model to a color shift detectionresult of the image data, the image processing apparatus including animage obtaining unit, a color shift detecting unit, a controlling unit,and a correction amount calculating unit.

The image obtaining unit obtains the image data.

The color shift detecting unit detects the color shift in this imagedata.

The controlling unit determines reliability of the fitting according tothe color shift detection result, and selects a magnification chromaticaberration model suitable for the color shift fitting.

The correction amount calculating unit performs the fitting of the colorshift detection result using the magnification chromatic aberrationmodel selected by the controlling unit and obtains correction amount ofthe magnification chromatic aberration for the image data based on thefitting result.

In an image processing apparatus, the controlling unit evaluates anddetermines the reliability in the fitting of the color shift detectionresult based on at least one of evaluation items including a detectionnumber of the color shift, a detection variation of the color shift, anda detection image height of the color shift. The controlling unitselects the magnification chromatic aberration model according to thisreliability.

In an image processing apparatus, the controlling unit limits a degreeof fitting freedom in the magnification chromatic aberration modelaccording to the color shift detection result.

In an image processing apparatus, the controlling unit has, for optionsof the magnification chromatic aberration model, at least (1) amagnification chromatic aberration model expressing nonlinearmagnification chromatic aberration against the image height and (2) amagnification chromatic aberration model expressing linear magnificationchromatic aberration against the image height.

In an image processing apparatus, the controlling unit selects differentmagnification chromatic aberration models for a center image area and aperipheral image area, respectively.

In an image processing apparatus, the controlling unit selects themagnification chromatic aberration model expressing the linearmagnification chromatic aberration against the image height for thecenter image area. The controlling unit selects the magnificationchromatic aberration model expressing the nonlinear magnificationchromatic aberration against the image height for the peripheral imagearea.

In an image processing apparatus, the controlling unit performsreliability determination of the color shift detection result for eachimage height range, and obtains an image height range satisfying apredetermined reliability condition, and sets the obtained image heightrange as the peripheral image area for using the nonlinear model.

In an image processing apparatus, the correction amount calculating unitfits the magnification chromatic aberration model to the color shiftdetection result by reducing an evaluation function e(Δ) of a fittingerror Δ between the color shift detection result and the magnificationchromatic aberration model. Here, this evaluation function e(Δ)satisfies the following formula for Δ1<Δ2 in a possible range of thefitting errors.e(Δ2)/e(Δ1)<(Δ2/Δ1)²  [1]

In an image processing apparatus, the correction amount calculating unitobtains correction amount of an isotropic magnification chromaticaberration (hereinafter, referred to as global correction amount) byfitting the magnification chromatic aberration model to the color shiftdetection result. Further, the correction amount calculating unitobtains an area adjustment value for adjusting errors between saidglobal correction amount and said color shift detection result for eachimage areas. The correction amount calculating unit obtains correctionamount of an anisotropic magnification chromatic aberration based on theglobal correction amount and the area adjustment values obtained in thismanner.

In an image processing apparatus, in the above described imageprocessing apparatus, the correction amount calculating unit obtains atwo-dimensional frequency map by classifying the color shift detectionresult to form frequency distributions for each image height. Thecorrection amount calculating unit obtains the correction amounts of themagnification chromatic aberration by fitting the magnificationchromatic aberration model to the two-dimensional frequency map.

An image processing program is a program causing a computer to functionas any one of the above described image processing apparatuses.

An electronic camera includes any one of the above described imageprocessing apparatuses and an imaging unit imaging an object to generateimage data. This image processing apparatus obtains correction amount ofthe magnification chromatic aberration for the image data generated inthe imaging unit.

An image processing method is a method performing the above describedprocessing in any one of the above described image processingapparatuses.

The image processing apparatus selects a magnification chromaticaberration model suitable for the fitting according to the color shiftdetection result. As a result, it becomes possible to reduce errors inestimating the magnification chromatic aberration from the color shifts.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an image processing device 11.

FIG. 2 is a flowchart illustrating operation of the image processingdevice 11.

FIG. 3 is a diagram showing a dividing pattern of an image plane.

FIG. 4 is a block diagram showing an electronic camera 21.

DETAILED DESCRIPTION OF THE EMBODIMENTS

<Description of a Configuration>

FIG. 1 is a block diagram of an image processing device 11.

In FIG. 1, the image processing device 11 is configured with thefollowing constituent requirements.

an image obtaining part 12 obtaining image data to be a detection targetof magnification chromatic aberration.

a color shift detecting part 13 detecting color shifts from the imagedata.

a controlling part 14 calculating reliability from a variation in acolor shift detection result or the like, and selecting a magnificationchromatic aberration model according to the reliability.

a correction amount calculating part 15 fitting the magnificationchromatic aberration model selected by the controlling part 14 to thecolor shift detection result and thereby obtaining correction amountsfor the magnification chromatic aberration of the image data.

Note that the above described constituent requirements may be realizedby software in which a computer executes an image processing program.Further, the above described constituent requirements may be realized byhardware such as an arithmetic circuit.

<Description of Operation>

FIG. 2 is a flowchart illustrating operation of the image processingdevice 11. The operation will be described with reference to thisflowchart.

Operation S1: The image obtaining part 12 reads each pixel (x, y) of aBayer image where RGB color components are arranged in a Bayer matrix.Here, a space of pixel coordinates is defined to be a righter side of animage plane for a larger x-coordinate and a lower side of the imageplane for a larger y-coordinate.

Operation S2: The color shift detecting part 13 performs interpolationprocessing on each of the RGB color components of the Bayer image, makesup dropped color components for pixel unit, and generates aninterpolated image having RGB color components in pixel unit.

Operation S3: The color shift detecting part 13 sets an optical axiscenter (cx, cy) of an imaging optical system in image plane coordinatesof the interpolated image. In a default setting, an image plane centerof the interpolated image is set to be this optical axis center. Notethat the color shift detecting part 13 may obtain coordinate values ofthe optical axis center (cx, cy) from added information (Exifinformation or the like) of an input image.

The color shift detecting part 13 divides the interpolated image in thecircumferential direction having a center at the optical axis center(cx, cy) as shown in FIG. 3, and sets a plurality of divided areas (N,NE, E, SE, S, SW, W, and NW shown in FIG. 3) and a radial direction foreach of the divided areas (arrow direction shown in FIG. 3).

Operation S4: Subsequently, the color shift detecting part 13 obtainsluminance Y of each pixel from the RGB color components of each pixel inthe interpolated image by a simplified calculation such as Y=(R+G+B)/3,for example. Then, the color shift detecting part 13 calculates a levelvariation grad Y(x, y) of the luminance Y along a radial direction in animage plane area where the magnification chromatic aberration can bevisually recognized (e.g., peripheral area apart form the optical axiscenter by a distance of 40% or more of a maximum image height hmax).

Here, by fixedly setting the radial direction in each of the dividedareas, it is possible to calculate the level variation along the radialdirection in a simplified manner according to the following formulas.Divided area N: grad Y(x,y)=Y(x,y−1)−Y(x,y)Divided area NE: grad Y(x,y)=Y(x+1,y−1)−Y(x,y)Divided area E: grad Y(x,y)=Y(x+1,y)−Y(x,y)Divided area SE: grad Y(x,y)=Y(x+1,y+1)−Y(x,y)Divided area S: grad Y(x,y)=Y(x,y+1)−Y(x,y)Divided area SW: grad Y(x,y)=Y(x−1,y+1)−Y(x,y)Divided area W: grad Y(x,y)=Y(x−1,y)−Y(x,y)Divided area NW: grad Y(x,y)=Y(x−1,y−1)−Y(x,y)

The color shift detecting part 13 searches a pixel position (x, y) wherean absolute value of the obtained level variation grad Y(x, y) is notless than a predetermined value Th (e.g., about ten levels in 256gradation data) for a level variation position along the radialdirection. Note that this search may be performed at appropriateintervals such as every six pixels, for example.

Operation S5: The color shift detecting part 13 performs range settingof a local window having a center at a position of the level variationposition detected at the operation S4. The color shift detecting part 13obtains an arrangement of the G-color components from this window.

Then, the color shift detecting part 13 obtains an arrangement of theR-color components from a range formed by shifting this window in aradial direction by a predetermined distance. The color shift detectingpart 13 performs level correction of the R-color arrangement such thatan average value of this R-color arrangement coincides with an averagevalue of the G-color arrangement. The color shift detecting part 13calculates a sum of absolute values of (G-color arrangement)−(R-colorarrangement after the level correction) to obtain an overlap error.

The color shift detecting part 13, while increasing or decreasing theshift distance of the R-color arrangement along the radial direction,searches a shift distance where the overlap error becomes a minimum toobtain a color shift of the level variation position. The color shiftdetecting part 13 temporally stores the obtained color shift togetherwith an image height value at the level variation position.

Note that it is preferable to obtain the color shift in a preciseness ofnot more than a pixel interval by interpolating the overlap error value.

Operation S6: The correction amount calculating part 15 generates atwo-dimensional frequency map by counting frequencies of the “imageheights and color shifts”, obtained at the operation S5 at each of thelevel variation positions for each of the divided areas.

For example, the two-dimensional frequency map divides the range of animage height h0 to the maximum image height hmax into 40 divisions and acolor shift range of −4 pixels to +4 pixels into 80 divisions, andcounts the frequency for each of the total 3,200 divisions.

Note that, each time the operation 5 detects a color shift, it ispreferable to update the two-dimensional frequency map of thecorresponding divided area, for saving a buffer capacity required forthe temporal storage of the “image heights and color shifts”.

Operation S7: The correction amount calculating part 15 generates atwo-dimensional frequency map of the entire perimeter of the image planeby combining the two-dimensional frequency maps of the respectivedivided areas. Further the correction amount calculating part 15generates a two-dimensional frequency map of the upper side of the imageplane by combining the two-dimensional frequency maps of the dividedareas N, NE, and NW, and generates a two-dimensional frequency map ofthe lower side of the image plane by combining the two-dimensionalfrequency maps of the divided areas S, SE, and SW, in the same process.Furthermore, the correction amount calculating part 15 generates atwo-dimensional frequency map of the right side of the image plane bycombining the two-dimensional frequency maps of the divided areas E, NE,and SE, and generates a two-dimensional frequency map of the left sideof the image plane by combining the two-dimensional frequency maps ofthe divided areas W, NW, and SW, in the same process.

Operation S8: The controlling part 14 sets a pitch ah to be about onetenth of the maximum image height hmax, and obtains:

-   n(h): the number of color shift detections in an image height range    h to h+Δh, and-   σ²(h): a deviation of color shifts in the image height range h to    h+Δh-   at predetermined intervals of the image height h (e.g., integer h).

Then, the controlling part 14 obtains reliability Ec of the color shiftdetection result by the following evaluation function, for example, foreach of the image height range.

$\begin{matrix}\left\lbrack {{Formula}\mspace{20mu} 1} \right\rbrack & \; \\{{{Ec}\left( {h\; 0} \right)} = {\sum\limits_{h = {h\; 0}}^{{h\;\max} - {\Delta\; h}}{\frac{n(h)}{\sigma^{2}(h)}\left( {h - {h\; 0}} \right)}}} & \lbrack 2\rbrack\end{matrix}$

This evaluation function [2] shows a larger value when the number of thecolor shift detection is larger in an image height range not lower thanan image height h 0. In addition, the evaluation function [2] shows alarger value when the variation of the color shift detection is smallerin the image height range not lower than the image height h 0. Further,the evaluation function [2] shows a larger value when the image height hof the color shift detection is higher in the image height range notlower than the image height h 0.

Operation S9: The controlling part 14 performs the followingclassification based on the reliability obtained at the operation S8 andselects a magnification chromatic aberration model.

(Case 1) Non-Linear Magnification Chromatic Aberration Model

The controlling part 14 determines whether or not there exists an imageheight h 0 satisfying three conditions: reliability Ec(h 0+1)<th 1, Ec(h 0)>th 1, and h 0>th_(linear).

Here, the threshold value th_(linear) is a threshold value showing anupper limit of the image height h where the magnification chromaticaberration changes linearly and determined preliminarily in anexperimental manner or in a statistical manner. For example, thethreshold value th_(linear) is set to be about a value of 4×Δh.

Meanwhile, the threshold value th 1 is a threshold value to determinewhether or not the reliability of the peripheral area of the image planeis suitable for estimating nonlinearity of the magnification chromaticaberration and determined preliminarily in an experimental manner or ina statistical manner. For example, the threshold value th 1 is set to beabout a value of 1,000.

If there exists an image height h 0 satisfying these three conditions,the image height h 0 is set to be a maximum value of a twist imageheight h_(twist) and the following nonlinear magnification chromaticaberration model f_(NL)(h) is selected.

$\begin{matrix}\left\lbrack {{Formula}\mspace{20mu} 2} \right\rbrack & \; \\{{f_{NL}(h)} = \left\{ \begin{matrix}{kh} & \left( {{{if}\mspace{14mu} h} < h_{twist}} \right) \\{{kh} + {a\left( {h - h_{twist}} \right)}^{2} - {\frac{a}{3\left( {{h\;\max} - h_{twist}} \right)}\left( {h - h_{twist}} \right)^{3}}} & \left( {{{if}\mspace{14mu} h} \geq h_{twist}} \right)\end{matrix} \right.} & \lbrack 3\rbrack\end{matrix}$

Note that unspecified parameters k, a, and h_(twist) in f_(NL)(h) willbe determined in the following operation S10.

(Case 2) Linear Magnification Chromatic Aberration Model

The controlling part 14 selects a linear magnification chromaticaberration model against the image height f_(L)(h)=k×h, if the threeconditions of the case 1 are not satisfied and Ec(th_(linear))>th 2 issatisfied.

Note that the threshold value th 2 is a threshold value determiningwhether or not the reliability of the peripheral area of the image planeis suitable for estimating linearity of the magnification chromaticaberration, and determined preliminarily in an experimental manner or ina statistical manner. For example, the threshold value th 2 is set to beabout a value of 400.

(Case 3) Discontinuance of Magnification Chromatic Aberration Estimation

The controlling part 14 determines that the color shift detection resulthas a low reliability and is not suitable for estimating themagnification chromatic aberration, when both of the case 1 and the case2 are not satisfied. In this case, the controlling part 14 discontinuesthe estimation and correction of the magnification chromatic aberration.

Operation S10: The correction amount calculating part 15 determines theunspecified parameters of the magnification chromatic aberration modelby fitting the magnification chromatic aberration model selected by thecontrolling part 14 to the color shift detection result of the entireperimeter of the image plane (two-dimensional frequency map of theentire perimeter of the image plane).

(Case of the Magnification Chromatic Aberration Model of Case 1)

The correction amount calculating part 15 performs initial setting ofdefault values close to correct solutions preliminarily obtained in astatistic manner to the unspecified parameters in the formula [3].

The correction amount calculating part 15 calculates a differencebetween a detected color shift value at a detection image height h and avalue of the formula [3] f_(NL)(h) at the detection image height h fromthe color shift detection result of the entire perimeter of the imageplane (two-dimensional frequency map) and obtains a fitting error Δ.

The correction amount calculating part 15 evaluates each of the fittingerrors Δ by the following evaluation function e(Δ).e(Δ)=|Δ|

The correction amount calculating part 15 sums up obtained values fromthe evaluation function e(Δ) to obtain a fitting evaluation value Ef.Note that, when using the two-dimensional frequency map, it is possibleto reduce a computation amount of the fitting evaluation values Ef bymultiplying the evaluation function e(Δ) obtained for each of thedivisions on the map by the number of the frequency in the division.

Further, the correction amount calculating part 15 obtains anonlinearity penalty term E_(NL) limiting a degree of freedom in thefitting by the following formula.E _(NL) =P _(NL) ×|a|×(hmax−h 0)

Here, P_(NL) is a coefficient determining a weight of the nonlinearitypenalty term and determined preliminarily in an experimental manner orin a statistical manner considering an occurrence degree of nonlinearestimation errors. For example, P_(NL) is set to be a value of about300. Meanwhile, h 0 is h 0 satisfying the above described threeconditions.

Note that, generally, the nonlinearity penalty term E_(NL) may be set tobe a small value when the unspecified parameter a has value within apredetermined range near zero and may be set to be a larger value whenthe value of the unspecified parameter goes out of the predeterminedrange.

The correction amount calculating part 15 determines the unspecifiedparameters k, a, and h_(twist) such that (Ef+E_(NL)) becomes to have aminimum or least value, while changing the unspecified parameters k, a,and h_(twist) in an appropriate pitch.

At this time, the unspecified parameter h_(twist) is limited to a valuenot more than the image height h0 which satisfies the three conditionsof the reliability. This limitation leads to a selection of thenonlinear magnification chromatic aberration model which shows thenonlinear magnification chromatic aberration against the image heightfor the peripheral area of the image having the image height not lessthan the image height h_(twist) which satisfies the conditions of thereliability.

(Case of the Magnification Chromatic Aberration Model of Case 2)

The correction amount calculating part 15 performs initial setting of adefault value close to a correct solution obtained preliminarily in astatistical manner to the unspecified parameter k in the linearmagnification chromatic aberration model f_(L)(h).

The correction amount calculating part 15 calculates a differencebetween a detected color shift value at a detection height h and a valueof f_(L)(h) at the detection height h from the color shift detectionresult of the entire perimeter of the image plane (two-dimensionalfrequency map) to obtain the fitting error Δ.

The correction amount calculating part 15 evaluates each of the fittingerrors Δ by the following evaluation function e(Δ).e(Δ)=|Δ|

The correction amount calculating part 15 sums up obtained values fromthe evaluation function e(Δ) to obtain the fitting evaluation value Ef.Note that, when using the two-dimensional frequency map, it is possibleto reduce a computation amount of the fitting evaluation values Ef bymultiplying the evaluation function e(Δ) obtained for each of thedivisions on the map by the number of the frequency in the division.

The correction amount calculating part 15 determines the unspecifiedparameter k such that the fitting evaluation value E f becomes to have aminimum or least value, while changing the unspecified parameter k in anappropriate pitch.

Operation 11: The correction amount calculating part 15 judgesvariations of the fitting error Δ in each of the divided areas anddetermines whether or not anisotropic adjustment of the magnificationchromatic aberration is performed.

If the variation is assumed to be within an allowable range of errors orthe like, the correction amount calculating part 15 does not perform theanisotropic adjustment and assigns zeros to area adjustment values ke,ks, kw, and kn for each area, and moves the process to an operation S13.

Meanwhile, if the variation exceeds the allowable range, the correctionamount calculating part 15 moves the process to an operation S12 toperform the anisotropic adjustment.

Operation S12: The correction amount calculating part 15 calculates thefollowing formula using the two-dimensional frequency map of the rightside of the image plane to obtain the area adjustment value ke of theright side area.ke=[sum of (color shift detection value at a detection image heighth−magnification chromatic aberration model value at the detection imageheight h)]/(sum of detection image heights h)

The correction amount calculating part 15 obtains the area adjustmentvalues ks, kw, and kn for the lower side area, left side area and upperside area, respectively, by similar calculations.

Operation S13: The correction amount calculating part 15 obtains areason an internal memory area for a correction amount map Δx(x, y) in thehorizontal direction and a correction amount map Δy(x, y) in thevertical direction. The correction amount calculating part 15 completesthese correction amount maps Δx and Δy by the following calculationprocessing.

First, the correction amount calculating part 15 calculates the imageheight h of coordinates (x, y) where a correction amount is to beobtained, using the coordinates of the optical axis center (cx, cy).h=[(x−cx)²+(y−cy)²]^(1/2)

Then, the correction amount calculating part 15 obtains Δx(x, y) foreach of the right side and the left side of the image plane according tothe following formulas, respectively.

$\begin{matrix}\left\lbrack {{Formula}\mspace{20mu} 3} \right\rbrack & \; \\{{\Delta\;{x\left( {x,y} \right)}} = \left\{ \begin{matrix}{\left\lbrack {\frac{f(h)}{h} + {kw}} \right\rbrack \times \left( {x - {cx}} \right)} & \left( {{{if}\mspace{14mu} x} < {cx}} \right) \\{\left\lbrack {\frac{f(h)}{h} + {ke}} \right\rbrack \times \left( {x - {cx}} \right)} & \left( {{{if}\mspace{14mu} x} \geq {cx}} \right)\end{matrix} \right.} & \lbrack 4\rbrack\end{matrix}$

Here, for f(h) in the formula, f_(NL)(h) is used for the case 1 andf_(L)(h) is used for the case 2.

The correction amount calculating part 15 obtains Δy(x, y) for each ofthe upper side and the lower side of the image plane according to thefollowing formulas, respectively, in the same process.

$\begin{matrix}\left\lbrack {{Formula}\mspace{20mu} 4} \right\rbrack & \; \\{{\Delta\;{y\left( {x,y} \right)}} = \left\{ \begin{matrix}{\left\lbrack {\frac{f(h)}{h} + {kn}} \right\rbrack \times \left( {y - {cy}} \right)} & \left( {{{if}\mspace{14mu} y} < {cy}} \right) \\{\left\lbrack {\frac{f(h)}{h} + {ks}} \right\rbrack \times \left( {y - {cy}} \right)} & \left( {{{if}\mspace{14mu} y} \geq {cy}} \right)\end{matrix} \right.} & \lbrack 5\rbrack\end{matrix}$

The correction amount maps Δx and Δy are completed by performing such acalculation sequentially for pixel unit except for the optical axiscenter.

Note that, as will be described below, when the magnification chromaticaberration correction is performed on R pixels of the Bayer image, thecorrection amount maps Δx and Δy may be generated only for pixelpositions (x, y) of the R pixels in the Bayer matrix. Besides, thecorrection amount map may be obtained for discrete sample points such asat each predetermined number of pixels (32 pixels or the like) and thecorrection amount between the sample points may be obtained byinterpolation. Further, the interpolation processing between thediscrete sample points may be omitted by using the same correctionamount for each block centering the discrete sample point. It ispreferable to speed up the computation processing of the correctionamount appropriately by such processing.

Operation S14: The correction amount calculating part 15 calculates ashift position (x′, y′) of the magnification chromatic aberration foreach coordinate position (x, y) of the R-color component in the Bayerimage by the following formula.x′=x+Δx(x,y)y′=y+Δy(x,y)

Then, the correction amount calculating part 15 obtains the R-colorcomponent at the shift position (x′, y′) by interpolation in the R-colorcomponents of the Bayer image. For this interpolation, a publicly knowninterpolation technique such as the cubic interpolation or the like canbe used. The correction amount calculating part 15 completes themagnification chromatic aberration correction for the R-color componentsof the Bayer image by replacing sequentially the R-color component atthe coordinates (x, y) by the interpolated R-color component at theshift position (x′, y′).

Operation S15: The image processing device 11 performs processingsimilar to the processing in the operations S5 to S14 also for B-colorcomponents of the Bayer image.

By the above described processing, it is possible to obtain thecorrection amounts of the magnification chromatic aberration and a Bayerimage after the magnification chromatic aberration correction.

<Advantages of the Embodiment and the Like>

The present embodiment selects the magnification chromatic aberrationmodel in which a fitting error is unlikely to occur, for the case 2having a low reliability in the color shift detection result. As aresult, it becomes possible to reduce the correction amount errors ofthe magnification chromatic aberration.

In addition, in the case 1 having a high reliability in the color shiftdetection result, the magnification chromatic aberration model which hasa high degree of freedom is selected for a more precise fitting of themagnification chromatic aberration. As a result, the correction amountsof the magnification chromatic aberration become more precise.

Further, the present embodiment judges the reliability of the colorshift detection result using the evaluation function, formula [2]. Thisformula [2] includes the following evaluation items.

-   -   Color shift detection number: A smaller detection number causes        a fitting shortage in the magnification chromatic aberration        model, resulting in a lower reliability.    -   Color shift detection variation: A larger detection variation        causes an imprecise fitting in the magnification chromatic        aberration model, resulting in a lower reliability.    -   Color shift detection image height: A smaller detection image        height provides a shorter color shift distance and causes a        lower detection preciseness of the color shift. Therefore, a        smaller detection image height causes a lower fitting        reliability.

By combining these three evaluation items, it becomes possible toprecisely evaluate the fitting reliability for the color shift detectionresult.

In addition, the present embodiment selects a linear magnificationchromatic aberration model for a center image area daringly even in thecase 1. Typically, the magnification chromatic aberration changes almostlinearly against the image height in the center image area. Therefore,by selecting the linear magnification chromatic aberration model for thecenter image area, it is possible to obtain correction values for anearly actual magnification chromatic aberration.

Further, the formula [3] of the present embodiment couples the twomagnification chromatic aberration models smoothly at the image heighth_(twist), and provides characteristics that a second-order derivativebecomes close to zero at a vicinity of the maximum image height hmaxwhere the color shift detection result is short. These characteristicsare close to that of the actual magnification chromatic aberration andthereby it is possible to obtain a correction value for a further nearlyactual magnification chromatic aberration.

In addition, the present embodiment obtains the image height range equalnot less than the image height h 0 which satisfies the three conditions,as an image height range satisfying the reliability in the case 1.h_(twist) is set so as to include this image height range, and thenonlinear magnification chromatic aberration model is selected for theperipheral image area having an image height not less than h_(twist). Bythis processing, the fitting reliability of the nonlinear magnificationchromatic aberration is secured and it becomes possible to reduce thecorrection amount errors in the peripheral image area.

Further, the present embodiment uses e(Δ)=|Δ| instead of the squareerror Δ² of the least square approach as an evaluation function of thefitting error Δ between the color shift detection result and themagnification chromatic aberration model. The least square approach canprovide a satisfactory fitting result when measurement errors follow thenormal distribution. However, the color shift detection errors includefactors caused by the image structure or false color and do not followthe normal distribution, and thereby a large measurement error is easilycaused frequently and abruptly. Therefore, in the least square approach,there arises a possibility that the fitting responds too sensitively tothis large measurement error and can not be performed correctly. On theother hand, the evaluation function e(Δ)=|Δ| in the present embodimentresponds not too sensitively to the large measurement error comparedwith the least square approach and it becomes possible to perform a moreappropriate fitting of the magnification chromatic aberration.

In addition, the present embodiment limits the degree of freedom of theunspecified parameters related to the nonlinearity by introducing thenonlinear penalty term. As a result, a variation of the unspecifiedparameter caused by the detection error is suppressed and it becomespossible to obtain a fitting of a magnification chromatic aberrationmodel based on a more actual case.

Further, the present embodiment obtains the global correction amount byfitting the isotropic magnification chromatic aberration model to thecolor shift detection result in the entire perimeter of the image plane.Furthermore, the area adjustment value reflecting the anisotropy of themagnification chromatic aberration is obtained from the comparisonbetween this global correction amount and the color shift detectionresult in each area. By combining this global correction amount and thearea adjusting value for each area, it is possible to obtain anappropriate correction amount of the anisotropic magnification chromaticaberration.

In addition, the present embodiment generates the two-dimensionalfrequency map by classifying the color shift detection results to formthe frequency distribution for each of the image heights. Generally, itis necessary to obtain the color shift detection results of more than ahundred to several thousands for calculating the correction amount ofthe magnification chromatic aberration precisely. Storing all of thesecolor shift detection results requires a large storage capacity.Besides, a computation amount becomes huge for calculating theevaluation values in the fitting processing. By using thetwo-dimensional frequency map, however, similar color shift detectionresults can be stored in the number of frequency and it becomes possibleto save the storage capacity for the color shift detection results. Inaddition, in the fitting processing using the two-dimensional frequencymap, it becomes possible to reduce considerably the computation amountof the fitting evaluation values by multiplying the evaluation functione(Δ) obtained for each division on the map by the frequency number inthe division.

<Supplements to the Embodiment>

Note that, in the foregoing embodiment, the fitting reliability isevaluated by the combination of the three evaluation items (detectionnumber, detection variation, and detection image height). However, theembodiment is not limited to this method. The fitting reliability may beevaluated by use of at least one of the three evaluation items.

In addition, the foregoing embodiment uses the fitting error evaluationfunction e(Δ)=|Δ|. However, the embodiment is not limited to thisfunction. An evaluation function e (Δ) satisfying the following formulamay be used for any Δ1<Δ2 in the possible range of the fitting error.e(Δ2)/e(Δ1)<(Δ2/Δ1)²  [1]Such an evaluation function is more resistant than the square error Δ²in the least square approach against the large measurement error, and itbecomes possible to perform a satisfactory fitting of the magnificationchromatic aberration model.

Further, the foregoing embodiment has described a single component ofthe image processing device 11. However, the embodiment is not limitedto the single component. For example, the image processing device 11 maybe mounted in an electronic camera 21 as shown in FIG. 4. In thiselectronic camera 21, the image processing device 11 can obtain thecorrection amounts of the magnification chromatic aberration for aphotographed image or a stored image in the imaging part 22 and furtherperform the magnification chromatic aberration correction.

Note that the foregoing embodiment uses the mathematical formula modelcoupling a linear expression and a cubic expression formula for thenonlinear magnification chromatic aberration model. However, the presentembodiment is not limited to this formula model. Generally, anynonlinear mathematical formula model may be used.

In addition, the foregoing embodiment estimates the magnificationchromatic aberration with the Bayer image as an object to be processed.However, the embodiment is not limited to this object to be processed.Generally, the present embodiment can be applied to any image as far asthe color shift is detected.

Note that the present embodiment can be implemented in other variousmanners without departing form the spirit and principal featuresthereof. Accordingly, the foregoing shows only an example in everyaspect thereof and should not be construed in a limiting sense. Thescope of the present embodiment is defined in the claims and is notlimited by the description in the specification in any way. Further, allthe modifications or variations in the range of the equivalents to theclaims fall within a scope of the present embodiment.

The many features and advantages of the embodiments are apparent fromthe detailed specification and, thus, it is intended by the appendedclaims to cover all such features and advantages of the embodiments thatfall within the true spirit and scope thereof. Further, since numerousmodifications and changes will readily occur to those skilled in theart, it is not desired to limit the inventive embodiments to the exactconstruction and operation illustrated and described, and accordinglyall suitable modifications and equivalents may be resorted to, fallingwithin the scope thereof.

1. An image processing apparatus that obtains a correction amount for ashift of position of a color component of image data due to amagnification chromatic aberration by fitting a magnification chromaticaberration model to a color shift detection result of the image data,the image processing apparatus comprising: an image obtaining deviceobtaining the image data; a color shift detecting device detecting colorshift in said image data; a controlling device determining reliabilityof said fitting according to said color shift detection result andselecting a magnification chromatic aberration model suitable for thefitting of said color shift; and a correction amount calculating deviceperforming the fitting of said color shift detection result using saidmagnification chromatic aberration model selected by said controllingdevice and obtaining the correction amount for the shift of position ofthe color component of the image due to the magnification chromaticaberration based on the fitting result, wherein said controlling devicehas, as options for said magnification chromatic aberration model, atleast a magnification chromatic aberration model expressing a nonlinearmagnification chromatic aberration against an image height, in which themagnification chromatic aberration is a function of the image height,and a magnification chromatic aberration model expressing a linearmagnification chromatic aberration against the image height.
 2. Theimage processing apparatus according to claim 1, wherein saidcontrolling device evaluates and determines a reliability in the fittingof said color shift detection result based on evaluation items includinga detection number of said color shift, a detection variation of saidcolor shift, and a detection image height of said color shift andselects said magnification chromatic aberration model according to saidreliability.
 3. The image processing apparatus according to claim 1,wherein said controlling unit: selects said magnification chromaticaberration model expressing the nonlinear magnification chromaticaberration against the image height, in which the magnificationchromatic aberration is a function of the image height, when thereliability of said fitting is higher than a predetermined standard; andselects said magnification chromatic aberration model expressing thelinear magnification chromatic aberration against the image height whenthe reliability of said fitting is lower than a predetermined standard.4. The image processing apparatus according to claim 1, wherein saidcontrolling device selects the magnification chromatic aberration modelexpressing the nonlinear magnification chromatic aberration against theimage height, in which the magnification chromatic aberration is afunction of the image height, a magnification chromatic aberration modelexpressing the linear magnification chromatic aberration against theimage height at a center image area and expressing the nonlinearmagnification chromatic aberration against the image height at aperipheral image area.
 5. The image processing apparatus according toclaim 4, wherein said controlling device performs a reliabilitydetermination on said color shift detection result for each image heightrange, and obtains an image height range satisfying a predeterminedreliability condition, and sets the obtained image height range as saidperipheral image area.
 6. The image processing apparatus according toclaim 1, wherein said correction amount calculating device fits saidmagnification chromatic aberration model to said color shift detectionresult by reducing an evaluation function e (Δ) of a fitting error Δbetween said color shift detection result and said magnificationchromatic aberration model, said evaluation function e (Δ) satisfyingthe following formula for any Δ1<Δ2 in a possible range of the fittingerrors e(Δ2)/e(Δ1)<(Δ2/Δ1)² . . . [1].
 7. The image processing apparatusaccording to claim 1, wherein said correction amount calculating deviceobtains a correction amount of isotropic magnification chromaticaberration, referred to as global correction amount, by fitting saidmagnification chromatic aberration model to said color shift detectionresult, obtains an area adjustment value for adjusting errors betweensaid global amount and said color shift detection result for imageareas, and obtains a correction amount of an anisotropic magnificationchromatic aberration based on said global correction amount and saidarea adjustment value.
 8. The image processing apparatus according toclaim 1, wherein said correction amount calculating device obtains atwo-dimensional frequency map by classifying said color shift detectionresult to form frequency distributions for each image height and obtainsthe correction amount of the magnification chromatic aberration byfitting said magnification chromatic aberration model to saidtwo-dimensional frequency map.
 9. A computer readable non-transitorymedium storing an image processing program causing a computer tofunction as an image processing apparatus according to claim
 1. 10. Anelectronic camera that obtains a correction amount for a shift ofposition of a color component of image data due to a magnificationchromatic aberration by fitting a magnification chromatic aberrationmodel to a color shift detection result of the image data generated bycapturing an image, the electronic camera comprising: an imaging deviceimaging an object to generate said image data; a color shift detectingdevice detecting a color shift in said image data; a controlling devicedetermining reliability of said fitting according to said color shiftdetection result and selecting a magnification chromatic aberrationmodel suitable for the fitting of said color shift; and a correctionamount calculating device performing the fitting of said color shiftdetection result using said magnification chromatic aberration modelselected by said controlling device and obtaining the correction amountfor the shift of position of the color component of the image due tomagnification chromatic aberration based on the fitting result, whereinsaid controlling device has, as options for said magnification chromaticaberration model, at least a magnification chromatic aberration modelexpressing a nonlinear magnification chromatic aberration against animage height, in which the magnification chromatic aberration is afunction of the image height, and a magnification chromatic aberrationmodel expressing a linear magnification chromatic aberration against theimage height.
 11. An image processing method performed by an imageprocessing apparatus that obtains a correction amount for a shift ofposition of a color component of image data due to a magnificationchromatic aberration by fitting a magnification chromatic aberrationmodel to a color shift detection result of the image data, the imageprocessing method comprising: an image obtaining operation obtaining theimage data by an image obtaining device; a color shift detectingoperation detecting a color shift in said image data; a controllingoperation determining reliability of said fitting according to saidcolor shift detection result and selecting a magnification chromaticaberration model suitable for the fitting of said color shift by acontrolling device; and a correction amount calculating operationperforming the fitting of said color shift detection result using saidmagnification chromatic aberration model selected at said controllingoperation and obtaining the correction for said image data for the shiftof position of the color component of the image due to the magnificationchromatic aberration based on the fitting result, wherein saidcontrolling device has, as options for said magnification chromaticaberration model, at least a magnification chromatic aberration modelexpressing a nonlinear magnification chromatic aberration against animage height, in which the magnification chromatic aberration is afunction of the image height, and a magnification chromatic aberrationmodel expressing a linear magnification chromatic aberration against theimage height.